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Abstract. Maritime safety and security require a constant surveillance of 
the traffic at sea, and several human actors are in charge of this control. 
Potential risks alerts, suspicious vessels and description of the situation are 
managed by visual surveillance systems. But these systems do not provide 
real analysis tools for maritime traffic data. Geovisualization and visual 
analytics have shown to be very efficient to handle big sets of heterogeneous 
data, and to make knowledge discovery easier to users. But advanced visual 
analytics environment can appear to be too complex to be used, according 
to user's skills. Surveillance tasks need to be formalized for having proper 
analysis tools. In our research toward user's skill adapted geovisualization, 
we propose here a study of visual surveillance tasks in maritime surveil- 
lance in a high-level taxonomy. Spatialization of the data will be processed 
according to these tasks, for proposing adequate visualization methods. 

Keywords: Visual analytics, geovisualization, maritime surveillance, risk 
management 



1 . Context & Objectives 

Usi ng the proper tools for leadi ng visual analytics of massive data is a prob- 
lem that has been raised by research for many years. In our research, we 
investigate the use of geovisual analytics for monitoring and analyzing traf- 
fic data in maritime domain. 

Maritime traffic is the most important traffic for merchandise exchange, 
representing more than 90% of world's trade. Its importance makes it a 
very sensitive domain, which needs a permanent monitoring. Safety and 
security have to be monitored and analyzed to prevent accidents or illegal 
activities at sea, such as pi racy, drug traffic or illegal immigration. 

Large vessels are tracked with the use of AIS (Automated Identification 
System), which broadcast kinematic and static information about the ves- 



sels by radio waves. The location, speed, heading, type of ship and other 
useful information are monitored with the use of Maritime Surveillance 
Systems (MSS). MSSs generally consist in monitoring screens with tables of 
data and a near real-time cartography of the monitored area. The scale of 
this area goes from a port, to a wi der area such as the M editerranean coast. 

Table 1 illustrates the type of data contained within an Al S message: as in- 
formation is manually registered, some fields can be missing or unreliable 
(e.g. the destination field here). Figure 1 gives an exampleof a visual system 
for vessels monitoring in a French MRCC (Maritime Rescue Coordination 
Center) of the Mediterranean Sea, CROSS-Med 1 in La Garde (France). 



Field 


Description 


Value example 


MMSI 


Maritime Mobile Service Identity 


236152040 


IMO 


International Maritime Organization identifier 


927851 


shipname 


Name of ship 


NIZZA LA BELLA 


country 


Country of immatriculation for ship 


France 


sestination 


Destination of ship 


HOME 


shiptype 


Type according to IMO 


Fishing 


shiplength 


Length in meters 


18 


shipdraught 


Draught / draft in meters 


3 


Ion 


Location of ship 


13.9751316 


lat 


Location of ship 


55.001736 


sog 


Speed in knots 


3.4 


rot 


Rate of turn 





heading 


Direction of ship in degrees 


125 


navstatus 


Description of navigation status 


Engaged_in_Fishing 


utctime 


Date and time of data 


2013-03-13 8:04:36.399 



Table L Exampleof data within an AIS message 



The major objective of these visual tools is to detect unusual or dangerous 
behavior at sea, that can be caused by human intentions (such as piracy), or 
uncontrolled events (material defects, meteorological context, etc.). Analy- 
sis of past events, that uses replay of the scene, is also a major task for mari- 
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time surveillance: analysts would look for the reason of an accident, or they 
would extract a pattern in illegal activities for further monitoring. 

However, various projects and studies showed that there is a need for im- 
proving the analysis of maritime data, for supporting analysts in their tasks 
(eg., Morel et al. 2010, Riveiro & Falkman 2011 Vatin & Napoli 2013). The 
major issue in current research is to improve the modeling of risks at sea, 
using machine-learning techniques (data-mining), semi -automated tech- 
niques (Spatial OLAP) or human-centered approaches (visual analytics) 
(Bedard, Rivest, & Proulx 2007, Idiri & Napoli 2012, Chaze et al. 2012). 
Although automated methods showed promising results in discovering pat- 
terns in accidents and traffic data, users prefer relying on visual methods, 
for there is not "black box" effect in the data analysis (Morel et al. 2010). 
Moreover, visual methods in data analysis allow users to understand the 
patterns easier, and remember it efficiently (Guo et al. 2011). 




Figure L Maritime surveillance system in a French MRCC (CROSS-Med) 



2. Visual Support for Traffic Data Analysis 

Geovisual analytics applied to mobile objects data have been well developed 
in the past few years. Traffic data are indeed widely used by the geovisual i- 
zation community for testing new algorithms and new visual tools (eg., 
Andrienko & Andrienko 2008, Andrienko & Andrienko 2011). In thefield of 
maritime monitoring, the recent works of Riveiro (2011) and Willems 



(2011) brought major improvements in visual analytics for anomaly detec- 
tion at sea. Their researches were based on Gaussian mixture models and 
advances Kernel density estimation for visualizing the behavior of ships at 
the scale of the sea, and compare it to "normal" kinematic and behaviors. 
The work of Hurter et. al (2009) also brought substantial i improvements i n 
the i nteracti on wi th traj ectori es data, appl i ed to ai r traff i c control . 

But the diversity of these works makes it even harder to choose the proper 
visualization methods for studied data. Moreover, previous researches in 
the 90's and 2000's have developed languages and decision tools for pro- 
posing adapted visualization according to the type of data (Shneiderman 
1996, Card & Mackinlay 1997). These studies showed that the process of 
visualization and the type of data to be analyzed have to be formalized, in 
order to be compared and used within an intelligent program: the most 
interesting models of the visualization process are the ones of Card & 
Mackinlay, then extended for dynamic data by Hurter & Conversy (2007), 
and the model of Chi (2000), for they describe the complete process of data 
modeling, filtering and visualizing. 

In our research, visualization is defined as the spatial metaphor of hetero- 
geneous data, such as described by Fabrikant & Skupin (2005): "spatializa- 
tion is defined as a data transformation method based on spatial meta- 
phors, with the aim of generating a cognitively adequate graphic repre- 
sentation for data exploration and knowledge discovery in multi- 
dimensional databases". This definition also represents the very process of 
data visualization described by Chi, and can be used in aformal description 
and programming language. This way, both geographical and non- 
geographical data benefit of the research in visualization, and can be com- 
bined in interactive visual interfaces. 

Figure 2 illustrates how Chi's Data State Reference Model is used to de- 
scribe the process of visualizing si mplemariti me geographical data, such as 
thelocation of vessels. Raw data (value) are the location of ships at the time 
of AIS emission ; analytical abstraction are pairs of decimal numbers for 
(Ion, lat) ; visualization abstraction isthespatialization into points; finally 
the view isthemap, using specific symbol ogy and interaction tools (such as 
pan, zoom, filter, etc.). The map in the figure represents the location of 
shi ps i n the N orth Sea ( north of the N etherl ands) on 15 M arch 11 15. 

Within an interactive map, or any other type of environment for visual ana- 
lytics, the visualized information can be concrete geographical data (such as 
ship location and trajectory, meteorology), concrete non-geographical in- 
formation (such as ship description) or abstract data (inferred information 
such as alerts, scenarios, data classification or data aggregation). Maritime 
surveillance requires the visualization of risk information, which Idiri & 



Napoli (2012) define as the combination of a ship's behavior, a geographic 
area (dangerous or not) and situation (such as ship type, or visibility). 




Figure 2. Data State Reference Model for visualization process taxonomy (based 
on Chi 2000), and the example of ship location mapping 



I n the next part, we go i nto detai Is about the specificities of risk control and 
its application to maritime surveillance with visual platforms. We describe 
the main within risks management and present an example of geovisualiza- 
ti on appl i ed to a specif i c mariti me event of i nterest. 



3. Formalization of Methods for Visual Surveillance 



3.1. High-Level of Visual Analytics Tasks 

Risk management process can be divided into four major steps, according 
to the work Wybo (2012), which are illustrated in Figure 3. This process is 
the base for discovering knowledge in the monitored domain, and for using 
this new knowledge efficiently. In the rest of this paper, we will explain 
what the corresponding tasks in maritime visual surveillance are, and the 
way geovisual analytics are used for each of these steps. This description 
hi ghl i ghts maj or steps that formal i ze vi sual anal yti cs tasks. 




Figure 3. Risk management process (based on Wybo 2012) 



Anticipation phase is the modeling of potential risks that are known and 
must be controlled and prevented. At step 2, vigilance means controlling 
the system of interest with a constant monitoring ; control is defined as the 
comparison between actual state of the system and intended (normal) state 
(Hollnagel & Woods 2005). Whenever unexpected events happen, an accu- 
rate analysis of the situation must be completed to take proper decisions 
(step 3), and to discover new rules for risk modeling (step 4). This general 
framework of risk management highlights the major concepts that can be 



completed with the use of (geo-) visual analytics: (1) characterization of 
risks and behaviors, (2) monitoring of the system of interest and (3) analy- 
sis of the events and data. 

Based on this general framework, we propose a high-level taxonomy of 
tasks for visual surveillance, since this is a major step toward the use of 
cognitively plausible visualization (Fabrikant & Skupin 2005). To describe 
thetasks of traffic surveillance, they are divided into the three concepts that 
were descri bed above. We apply these steps to maritime surveillance by the 
means of (geo-) visual analytics. First, the identification and description of 
risks and dangerous behaviors at sea: this is the baseline used for control- 
ling risks and comparing monitored data to models of risky situations. 
Then, the monitoring tasks are the most common uses of maritimesurveil- 
lance systems and anomalies detection (unusual behavior) in near real-time 
data. Finally, the analysis tasks: these are more advanced tasks that take 
into account historical data, in order to detect patterns and to understand 
the conditions of an event (such as collision, grounding). Table 2 lists these 
high-level tasks and gives some examples for each concept. 



Task 


Description 


Geovisual analytics examples 


Identification 


Modelling of behaviors or areas 
that present potential risks. 


Mapping sensitive zones regarding 
piracy or meteorology. 
Visualizing dangerous profile of ships to 
look for. 


Monitoring 


Near real-time surveillance of 
the information, and decision- 
making. 


Detecting fishing ship entering a 
restreint fishing area. 
Detect a stop at sea. 


Analyzing 


Manipulation of historical data to 
extract patterns and knowledge. 


Comparing usual fishing behavior to a 
specific ship's behavior. 
Searching for crossing or close 
trajectories. 



Table 2. High-level tasks for visualization in surveillance system 



The low- level tasks are specifications of these high-level tasks: they would 
depend on the type of risk to monitor and the information to take into ac- 
count. To go deeper into this taxonomy, the concepts to which thetasks are 
applied are classified within four major categories: objects, behaviors, 
events and context. Some exampl es i n Tabl e 2 i 1 1 ustrate these categori es. I n 
the next part, we describe a specific event of fishing alert to illustrate the 
principle of interaction between objects, behaviors, events and context in- 
formation, and we explain how visualization can be used. 



3.2. Visual Detection of a Maritime Fishing Alert 

These general tasks for visual surveillance need to be specified with low- 
level concepts. This way, the properly adapted geovisualization methods 
could be proposed. As our study aims at developing a methodology for pro- 
posing plausible visual methods, these specific concepts must concern visu- 
al tasks such as visual mining and communication. For this purpose, we use 
a bottom- up approach for risks detection and analysis: starting from well- 
known risks or forbidden ship behaviors, we describe the visual tasks that 
have to be performed. 

Some examples of major issues that have to be recognized are: prohibited 
fishing, unusual trajectory, drug traffic or determined attack (such as pira- 
cy) . To i nvesti gate the contri buti on of geovi sual anal yti cs, we wi 1 1 gi ve more 
details about prohibited fishing events. Prohibited fishing can be caused by 
a fishing ship stopping in a restricted fishing area, two fishing ships having 
parallel trajectories (parallel fishing) or two fishing ships close to each other 
and stopping at sea. Vandecasteele& Napoli (2012) modeled these types of 
risks within an ontology, for automated alerts discovery. We use this model 
of risks for investigating the needs in visualization. Figure 4 illustrates the 
ontological model for illegal fishing scenario, to be used in an automated 
process. Each of the concerned objects requires a corresponding visual 
modeling to be discovered by visualization instead of automation. 




Figure 4. Model of illegal fishing scenario within an ontology (based on Vandeca- 
steele& Napoli 2012) 



Modeling these events with visual features can goes from simplegeovisuali- 
zation of geographic concrete features to more advanced geovisual analytics 
of geographical and abstraction space, spatialized within a visualization. 

Figure 5 illustrates a simple illegal fishing scenario, but only some charac- 
teristics of previous model are visualized: the ship type is Fishing (colored 
legend) and it issituated inside an area of restricted fishing. As this visuali- 
zation is a "snapshot" at a certain time, information about its kinematic are 
not displayed. Yet, information about low speed or the shape of its trajecto- 
ry has to be visualized to get real knowledge on the situation. An accurate 
assessment of possible infraction cannot be made at this point. Automation 
can raise an alert from previous ontology, by processing information found 
within the database, but visualization has to be improved if used by human 
operators without artificial intelligence. 




To improve the visualization, Figure 6 displays the past trajectory of the 
ship, with color code for its speed (in knots in the legend) and the current 
speed with a proportional line. These simple encodings of the information 
can help comparing actual situation to a potential illegal behavior, which 
was previously characterized (global body of knowledge). In this case, we 



can visually extrapolate the future trajectory of the ship, and understand 
there i s no i nf racti on yet, and there i s I i ttl e chance the shi p wi 1 1 stop now. 




Figure 6. The same fishing ship with extra information on speed and trajectory 



For the same type of scenario, a particular information about the kinematic 
of the shi p i s the stop event. Detecti ng a stop i n thi s type of 2D representa- 
tion of a trajectory is not possible, as a stop is not a line. Two means could 
be proposed for stop detection. First, using a 3D space-time cube allows to 
detect the speed of movi ng obj ect by the si ope of the I i ne: weak si ope means 
the object traveled a large distance during a few time (high speed), whereas 
strong slope means the object made short travel during a long time (low 
speed). But visualizing both dynamic of vessel in 3D and context infor- 
mation (map of restricted area) is quite complicated in 2D. Second solution 
is to visualize stop as a point, of which one attribute could be the duration 
of spot. This way, a new visual variable is used for visualizing this event: for 
instance, a circle with varying size or color. 

Figure 7 illustrates a type of visualization of stops at sea, where size and 
color of circle stand for the duration of stop. In this representation of aspe- 
cific part of a trajectory a large red circle stands for stop of more than lh, 
completed by extra information on demand. This type of geovisualization, 
synchronized with a speed graph, is easier for searching long stops at spe- 
cific areas as stops and areas limits are on the same plan. I n this example, 



speed graph shows a long stop, but the map visualization explains this stop 
was in a port area. 




Figure 7. Trajectory, speed and stop information 



F rom th i s exampl e of a possi bl e f i sh i ng al ert, some categor i es of i nterest for 
visual analytics of risk at sea can be extracted. The ontology developed by 
Vandecasteele (2012) and the results from data-mining by Idiri & Napoli 
(2012) proved that risks can be described by the interaction between: geo- 
metrical data (monitored objects), context data, time data (interval of inter- 
est), kinematic data and attributes of objects. For visualization purpose, 
each of these i nf ormati on I evel s has to be taken i nto account i n the geovi su- 
al analytics environment as we have seen in the example above. Within a 
human-machine environment, three major spaces for visualization require 
synchronization for their analysis: map space, attribute space and time 
space. I n these spaces, the same information can be represented more than 
once i n order to extract patterns, such as ti me and geographi cal context. 

The development of low-level tasks must take into account the type of rele- 
vant information (time, space, attribute, etc.), the amount of data and their 
extent ( i n ti me or space) . 

Table 3 summarizes the information spaces and the process of visualization 
that was chosen for this geovisualization example. Chi's Data State Model 
for the visualization process has been used to describe various process for 



obtaining final geovisualization. Arrows (—0 are used when data are already 
in a proper format for visualization. 





Within Value 


Data Transfor- 
mation 


Within Analytical 
Abstraction 




Context 


Restricted area 
shape 








Object 


Vessel position; 
Vessel trajectory 


Extract coordinates 
and relative data 


Dynamic filtering 


b 


Event 


Stop 


Extract position and 
stop duration 








Vessel past speed 


Extract past records 








Vessel speed 


Extract speed 








Vessel type 









Visualization 
Transformation 


Within Visualiza- 
tion Abstraction 


Visual Mapping 
Transformation 


Within View 




Choose color code 




Map area 


Pan, zoom 




Create 2D points 
and lines 


Abstraction: points; 
lines 


Map point & line 
features 


Pan, zoom, details 
on demand 




Create circle 


Abstraction: colored 
circle, various size 


Map point features 


Pan, zoom, details 
on demand 




Create time series 


Abstraction: time 
series of speed 
values 


Map into line chart 


Filter, zoom 




Create vector 


Abstraction: line 


Map speed vector 






Choose color for 
type 


Color 


Use on location 
point 





Table 3. Data State Model applied to restricted fishing example 



4. Conclusion and Future Work 

I n this paper, we investigated the major role of visualization for traffic con- 
trol, especially the maritime domain. We have seen there is a need for for- 
malizing of the various visual tasks of control and the events that happen at 
sea, in order to propose the most adequate visual analytics tools. The in- 
formation spaces of (1) space, (2) time and (3) attributes must be visualized 
and controlled by interactive tools for data exploration. Knowing the type of 
analysisto belead and the behaviors and events to find will help in building 
proper visualization environment. 



As discussed in the previous parts, visual analytics understanding and effec- 
tive use strongly depends on user's ski I Is. If the user does not understand a 
method of visualization, it won't be properly used, or even not used at all. 
That's why user's evaluation of geovisualization must betaken into account, 
to propose most usable geovisualizations. Future work will concern (1) the 
extension of low- level tasks and (2) evaluating the way user perceives easi- 
ness of use what the main features for understanding visualizations are 
(amount of data, animation, etc.). For this purpose, we will use evaluations 
indicators such as Perceived Usefulness and Perceived Ease of Use de- 
scribed by Davis (1993) in theTechnology Acceptance Model. 
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